Methods and systems for searchlight control for aerial vehicles

ABSTRACT

Systems and methods described herein provide one or more processors configured to execute program instructions to cause the one or more processors to receive camera images of a scene including a portion being lit by the searchlight, receive terrain data from a terrain database, generate an augmented/virtual reality scene based on the camera images and the terrain data, display, via a headset display, the augmented/virtual reality scene, track gaze and head movement using headset sensors, output tracking data based on tracked gaze and head movement, and output instructions for controlling at least one function of the searchlight based on the tracking data.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims benefit of prior filed Indian ProvisionalPatent Application No. 202011006205, filed Feb. 13, 2020, which ishereby incorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure generally relates to controlling searchlights foraerial vehicles.

BACKGROUND

Search and Rescue (SAR) operations are often performed using aircraftwith specialized rescue teams and equipment to assist people in real orlikely distress. These include mountain rescue, ground search andrescue, air-sea rescue, etc. Generally, rotary wing or fixed wingaircraft are used for aerial SAR operations.

Systems like InfraRed (IR) cameras and Forward Looking IR Radar (FLIR),which detect thermal radiation, and night vision goggles are used inperforming SAR during night operations.

The night search and rescue is a challenging task. Either the On SceneCoordinator (OSC) or the pilot will have considerable workload whenperforming SAR operations during night time. Among multiple factors thatgovern the effectiveness of SAR during night time, illumination fromsearch, navigation and other lights play a considerable role.

Operations performed by the coordinator include stow, deploy, filterchange, zoom, searchlight dim along with pointing-scanning andcoordinating. Available systems include hand grip controllers,searchlight control panels and filter selectors that interface with asearchlight to carry out these activities. Such interfaces with thesearchlight are quite cumbersome and put a considerable amount of loadon the person conducting the search. Existing hand grip controllerslimit a degree of freedom of movement of a search coordinator or pilot,which can limit the effectiveness of the search operation.

Accordingly, it is desirable to provide methods and systems to improveuser interfaces with search and rescue equipment including searchlightcontrol. In addition, it is desirable to provide user interfaces thatallow a user to wholly focus on the search tasks and to minimizedistraction of how to operate control interfaces. Furthermore, it isdesired to reduce workload of a person conducting search operations andto facilitate focus on executing the mission. Furthermore, otherdesirable features and characteristics will become apparent from thesubsequent detailed description and the appended claims, taken inconjunction with the accompanying drawings and the foregoing technicalfield and background.

BRIEF SUMMARY

Systems and methods described herein provide one or more processorsconfigured to execute program instructions to cause the one or moreprocessors to receive camera images of a scene including a portion beinglit by a searchlight, receive terrain data from a terrain database,generate an augmented/virtual reality scene based on the camera imagesand the terrain data, display, via a headset display, theaugmented/virtual reality scene, track gaze and head movement usingheadset sensors, output tracking data based on tracked gaze and headmovement, and output instructions for controlling at least one functionof the searchlight based on the tracking data.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the subject matter may be derived byreferring to the detailed description and claims when considered inconjunction with the following figures, wherein like reference numbersrefer to similar elements throughout the figures.

FIG. 1 illustrates a block diagram of a searchlight control system, inaccordance with embodiments of the present disclosure; and

FIG. 2 illustrates a flowchart of a method of operating the searchlightcontrol system of FIG. 1 , in accordance with embodiments of the presentdisclosure.

DETAILED DESCRIPTION

The following detailed description is merely illustrative in nature andis not intended to limit the embodiments of the subject matter or theapplication and uses of such embodiments. As used herein, the word“exemplary” means “serving as an example, instance, or illustration.”Any implementation described herein as exemplary is not necessarily tobe construed as preferred or advantageous over other implementations.Furthermore, there is no intention to be bound by any expressed orimplied theory presented in the preceding technical field, background,brief summary or the following detailed description.

Techniques and technologies may be described herein in terms offunctional and/or logical block components and/or modules, and withreference to symbolic representations of operations, processing tasks,and functions that may be performed by various computing components ordevices. Such operations, tasks, and functions are sometimes referred toas being computer-executed, computerized, software-implemented, orcomputer-implemented. It should be appreciated that the various blockcomponents and modules shown in the figures may be realized by anynumber of hardware, software, and/or firmware components configured toperform the specified functions. For example, an embodiment of a systemor a component may employ various integrated circuit components, e.g.,memory elements, digital signal processing elements, logic elements,look-up tables, or the like, which may carry out a variety of functionsunder the control of one or more microprocessors or other controldevices.

When implemented in software or firmware, various elements of thesystems described herein are essentially the code segments orinstructions that perform the various tasks. In certain embodiments, theprogram or code segments or programming instructions are stored in atangible processor-readable medium, which may include any medium thatcan store or transfer information. Examples of a non-transitory andprocessor-readable medium include an electronic circuit, a semiconductormemory device, a ROM, a flash memory, an erasable ROM (EROM), a floppydiskette, a CD-ROM, an optical disk, a hard disk, or the like.

Disclosed herein is a searchlight control system offering improvementsin searchlight technology. The present disclosure proposes usage ofmultimodal interactions with the searchlight to provide superiorperformance and increase overall mission efficiency. The disclosedsearchlight control system is interfaced with multimodal controlinteractions, which can lead to more autonomous mission operations andreduce previous laborious steps for a pilot or mission coordinator. Thepresent disclosure describes a head mounted display that displays asearch scene captured by a camera mounted to an aerial vehicle,processes the captured images to provide an augmented or virtual realitydisplay and utilizes gaze tracking as one input for controlling asearchlight. In embodiments, additional inputs include speechrecognition for controlling at least one function of the searchlight.

Systems and methods described herein reduce the load of a personconducting search operations and facilitate focus on executing themission. The searchlight control system enables a searchlight connectedwith multiple modalities to improve accessibility and efficiency of thesearch mission. The multiple modalities include voice and/or gesturecontrol in conjunction with an augmented display that will aid thecoordinator in accomplishing the mission with ease. The herein describedsearchlight control system allows a greater degree of freedom ofmovement for an on-scene coordinator, improved focus on mission aspects,enhanced situational awareness and facilitates quick decision making.

Referring to FIG. 1 , remote video from a camera 46 mounted to an aerialvehicle 40 is streamed to a headset 14 including a headset display 16via an augmented/virtual reality generation module 64 that includesregistered synthetic features in a display. Headset motion sensors 18and a headset gaze camera 20 allow gaze and head tracking data 72 to bedetermined, which represents where a search coordinator is looking inthe augmented scene. Searchlight controller 42 is responsive to the gazeand head tracking data 72 to control searchlight 12 (e.g. by changingfocus and/or changing beam direction) to provide required additionalillumination at a location in real space corresponding to where thecoordinator is detected to be looking in image space. Thus, headset 14is configured to detect a gaze area of the coordinator and perform gazeslaving to adjust a direction of beam 60 from searchlight to the areafor better illumination. Further, a perspective immersive view isgenerated in headset display 16 using augmented reality to enhancesearch operation. Augmented/virtual reality generation module 64 isconfigured to augment live video using registered synthetic terrainfeatures based on terrain data 58 from a terrain database 38. Further,augmented/virtual reality generation module 64 is configured to performobject recognition (using other database sources) and augmentationobject tracking (based on coordinator voice or gesture command).

In embodiments, searchlight control system 10 additionally includesvoice recognition unit 30 to provide a further mode of input for thesearchlight control system. Having a voice activated search light allowsgreater degree of freedom of movement of a searchlight coordinator.Microphone 88 of headset 14 is either wirelessly or connected directlyto voice recognition module 32 or a separate microphone 88 is provided.Voice recognition module 32 is trained for a list of possible search andrescue vocabulary. This vocabulary includes search light movementcommands (such as turn, tilt, move, rotate), searchlight propertycommands (such as increase/reduce brightness, focus, slave to camera),filter selection commands and tracking commands (such as track object(human/vehicle) in focus, record). In some embodiments, gaze slavingcontrol for direction of searchlight 12 is combined with voice activatedcontrol of other searchlight features like searchlight propertycommands, filter selection and/or tracking commands. Voice recognitionmodule 30 generates voice control commands 90 for the given speechcommand and searchlight controller 42 is configured to react to voicecontrol commands 90 to control searchlight 12.

In embodiments, searchlight control system 10 includes a display device24 as an additional control input. Display device 24 is configured todisplay output from camera 46 remotely in a monitor/handheld system likea tablet. Display device 24 is configured to allow synchronous operationof viewing and searching in poor weather/visibility conditions. That is,live images of the camera 46 can be viewed and touchscreen controls canbe used to adjust searchlight functions (like beam direction,searchlight property, filter selection and tacking). In this way, acontrol-on-the-go process is provided by which a searchlight 12 can beadjusted while viewing the scene 52. In embodiments, a currentstate/orientation of the searchlight 12 is sent from searchlightcontroller 12 to display device 24. The view stream is coupled with atouchscreen interface provided for the coordinator to re-direct thesearchlight 12 and includes a set of soft switches to change theorientation of the searchlight 12 and change the light parameters bysending corresponding commands to the searchlight controller 12. Controlof the searchlight 12 may be passed between headset 14 and displaydevice 24 based on mode selections and may be distributed between themso that some searchlight functions are controlled by headset 14 and someby display device 24. Display device 24 can also work in conjunctionwith voice activated searchlight 12, wherein the coordinator can usevoice to control some searchlight functions (e.g. tracking) and somesearchlight functions are controlled through display device 24 (e.g.searchlight property) and some searchlight functions are controlledthrough headset 14 (e.g. gaze slaved beam direction). In embodiments,display device 24 includes an interactive map module 26 providing agraphical map allowing touch control to select areas of interest and togenerate searchlight and/or aerial vehicle commands to move the aerialvehicle 40 and the searchlight 12 to the area of interest throughautopilot system 80 and searchlight controller 42. In some embodiments,interactive map is displayed in augmented/virtual reality world ofheadset 14 and similar area of interest selection can be made by gesturecontrol which is detected by gesture determination module 22.

Having summarized in the foregoing overall functions of searchlightcontrol system 10, more detailed information will now be provided withreference to FIG. 1 . FIG. 1 illustrates a searchlight control system 10that includes a searchlight 12, a headset 14 including headset motionsensors 18 and a processing system 36 including at least one processor48 configured to execute instruction of a computer program 50. Accordingto the computer program instructions, the processing system 36 isconfigured to receive image data 56 from a camera 46 (e.g. video datafrom camera 46) of a search scene 52 including a portion being lit by abeam of searchlight 12. Camera 46 and searchlight 12 are mounted to anaerial vehicle 40. Camera 46 and searchlight 12 are useful in a searchand rescue mission or other mission in which searchlight 12 needs to beaccurately controlled in order to improve visibility and identify aground target. Aerial vehicle 40 is, in some embodiments, a rotor craftsuch as a helicopter and other types of Vertical Take-Off and Landing(VTOL) aerial vehicle. Manned or unmanned aerial vehicles haveapplication according to the present disclosure. Searchlight 12 is, inone embodiment, an apparatus that combines an extremely bright source(e.g. at least 10,000 lumens) with a mirrored parabolic reflector toproject a powerful beam of light of approximately parallel rays in aparticular direction. Searchlight 12 is configured to allow swiveling soas to direct the beam 60 to different positions on the ground.Multi-axis swiveling is possible and is effected through searchlightactuators 62 such as electromotors under control based on searchlightcontrol data 64 from searchlight controller 42.

In embodiments, camera 46 includes one or more cameras that operate(primarily) in the visible spectrum or in the infrared spectrum or acombination thereof. In embodiments, camera 46 is part of an EnhancedVision System (EVS). An EVS camera is an airborne system that captures aforward-looking scene so as to provide a display that is better thanunaided human vision. EVS camera includes imaging sensors (one or more)such as a color camera and an infrared camera or radar. EVS cameraincludes, in embodiments, a millimeter wave radar (MMW) based imagingdevice, a visible low light television camera, one or more InfraRedcameras (possibly including more than one infrared camera operating atdiffering infrared wavelength ranges) and any combination thereof toallow sufficient imaging in poor visibility conditions (e.g. because ofnight time operation or because of inclement weather).

The processing system 36 is configured to receive terrain data 58 from aterrain database 38. Terrain database 38 includes data elementsdescribing ground terrain and some buildings. Thus, slopes, hills,mountains, buildings and even trees can be described in terrain database38. Terrain database 38 allows terrain features to be included indisplays generated by processing system 36, particularlythree-dimensional perspective displays provided through headset display16 of headset 14.

The processing system 36 is configured, via augmented/virtual realitygeneration module 64, to generate an augmented/virtual reality scenebased on image data 56 from camera 46 and terrain data 58 from theterrain database 38. Other synthetic feature data sources in addition tocamera 46 and terrain database 58 can be provided. In embodiments,augmented/virtual reality generation module 64 is configured to receiveat least image data 56 from camera 46, position and orientation data 66and terrain data 58. Position and orientation data 66 defines positionand orientation of camera 46 and camera parameters in order to be ableto localize the scene 52 being viewed by the camera 46. Position andorientation data 66 is generated based on global positioning andorientation data obtained from sensor system 68 of aerial vehicle 40.Sensor system 68 includes a global positioning receiver that determinesposition of aerial vehicle 40 based on satellite signals received fromat least three satellites. Further, sensor system 68 includes anInertial Measurement Unit (IMU) to allow orientation of the aerialvehicle (yaw, pitch and roll) to be determined. Yet further, sensorsystem 68 includes sensors (optionally associated with searchlightactuators 62) to allow relative orientation and position of camera andaerial vehicle 40 to be determined. Other camera parameters includingzoom level may be determined in sensor system 68. The position andorientation data 66 from sensor system 68 is fused by augmented/virtualreality generation module to precisely localize the scene being capturedby the camera 46 in real word space. Position and orientation data 66can be used to positionally register synthetic terrain features definedin terrain data 58 (and optionally other synthetic features from otherdata sources) using known transformation functions between real spaceand image space, thereby generating an augmented or virtual realityscene. Augmented/virtual reality generation module 64 is configured togenerate display data 70 representing the generated augmented/virtualreality scene. Thus, a live feed from camera 46 is perfectly registeredvia position and orientation data 66 on top of a 3-D graphical view ofthe terrain (and possibly other synthetic features), creating a blendedimage that gives pilots and search coordinators enhanced situationalawareness. Augmented reality adds digital elements from terrain database38 and other data sources to a live view from camera 46. Virtual realityimplies a complete immersion experience such that live view from camera46 is wholly replaced by corresponding synthetic features. A virtualreality view may be intermittent to allow an operative greatervisibility when orienting in the search scene 52 before reverting toaugmented reality to locate a potential target in the scene 52.

The headset display 16 of headset 14 is configured to display thevirtual/augmented reality scene based on the display data 70 generatedby augmented/virtual reality generation module 64. Headset display 16includes a stereoscopic head-mounted display that provides separateimages for each eye.

Headset 14 includes headset motion sensors 18 configured to track headmovement. Headset motion sensors 18 include an IMU including one or moreaccelerometers and gyroscopes to allow position and orientation (yaw,pitch and roll) of the head to be calculated. In some embodiments,headset 14 includes an outwardly facing headset camera (not shown) or acluster of ranging devices to allow for highly accurate determination ofposition and orientation of headset 14 through Simultaneous LocalizationAnd Mapping (SLAM) techniques. Headset motion sensors 18 are configuredto output position and orientation data as part of gaze and headtracking data 72. In some embodiments, headset 14 includes one or moreinwardly facing headset gaze cameras 20 to allow eye gaze of a wearer tobe tracked according to known techniques. Headset 14 includes an eyetracking module that receives images of the eyes from headset gazecamera (s) 20 and outputs an eye tracking vector as part of gaze andhead tracking data 72. Accordingly, headset 14 is configured to generategaze and head movement tracking data 72 that is used to control avariety of features of searchlight control system 10.

Searchlight control system 10 includes a searchlight controller 42 thatis configured to output searchlight control data 64 for controlling atleast one function of the searchlight 12 based on the gaze and headmovement tracking data 72. In embodiments, eye tracking vector (which ispart of gaze and head movement tracking data 72) includes dataconcerning a gaze point, fixations, amongst other gaze metrics. Inembodiments, searchlight controller 12 is configured to gaze slavemovement of the searchlight 12 and to gaze slave control movement of thesearchlight 12 through searchlight actuators 62 and searchlight controldata 64. That is, searchlight controller 12 is configured to determine apoint or area of interest according to the gaze of the wearer of theheadset 14 as defined in the gaze vector, to transform that point orarea of interest in image space into real word space (using an inverseof a similar transform to that described above with respect toaugmented/virtual reality generation module 64) and to generatesearchlight control data 64 accordingly. In some embodiments, gaze andhead movement tracking data 72 is utilized by augmented/virtual realitygeneration module 64 to execute foveated rendering by rendering parts ofthe scene 52 that a wearer is looking at in higher resolution thanperipheral parts.

In accordance with various embodiments, position and orientation datarepresenting head movements of a user are output from headset 14 as partof gaze and head movement tracking data 72, which is used as a controlinput for augmented/virtual reality control module 64. Specifically, asthe head orientation moves (e.g. pitch, roll and/or yaw), this movementis captured in gaze and head movement tracking data 72 and part of thescene 52 that is displayed should be updated accordingly.Augmented/virtual reality generation module 64 is configured to generatedisplay data 70 to show part of the scene 52 according to head movementsof a wearer of headset 14 (as defined in gaze and head movement trackingdata 72). This tracking of head movement to the part of scene 52 to bedisplayed is determined based on a transform of a view of the wearer inimage space (which is known from headset display parameters andorientation of headset 14 included in gaze and head movement trackingdata 72) into real word space of the scene 52 (which is known based onposition and orientation data 56 including global position of aerialvehicle 40, orientation of aerial vehicle 40, orientation of camera 46and other parameters) and the augmented/virtual reality generationmodule 64 generates image data 70 spatially corresponding to therequired part of the scene 52. When head movement indicated by gaze andhead movement tracking data 72 is outside of current field of view ofcamera 46, processing system 36 is configured to generate scene capturedata 76 defining an area of scene 52 that is to be captured (e.g. adefined geographical area). Based on scene capture data 76, cameracontroller 46 of aerial vehicle 40 is configured to generate cameracontrol data 78, which is used by camera actuators 74 to capture thearea of scene 52 defined by scene capture data 76.

In some embodiments, searchlight control data 64 and/or camera controldata 78 is useful for automatically controlling though an autopilotsystem 80 a position of the aerial vehicle 40 as well as, oralternatively to, utilizing searchlight actuators 62 and/or cameraactuators 74 to control position and orientation of searchlight 12and/or to control position and orientation of camera 46.

The present disclosure provides a searchlight control system 10 thatallows a wearer of headset 14 to control direction and orientation ofbeam 60 by where the wearer is looking in a display of scene shown onheadset display 18. An augmented or virtual reality view is providedthrough headset display 16 that allows the wearer to be fully immersedin the search mission and to intuitively control the beam of thesearchlight 12. Further, which parts of scene 52 are being viewed iscontrolled by head movement (and optionally additionally eye tracking),which can be used to control aim of camera 46. Thus, the scene 52 ismore clearly viewed and the searchlight 64 can be more effectivelycontrolled than with prior art control systems.

Searchlight 12 is controlled by additional control inputs to gazeslaving, in accordance with various embodiments. In embodiments, headset14 includes a gesture determination module 22 configured to determinehand gestures of a wearer of headset 14. In some embodiments,searchlight control system 14 includes hand controllers (not shown) thathave light emitting devices to that allow gesture determination module22 to determine position and orientation of the hands throughconstellation tracking methods. In other embodiments, hand controllersare not required and headset 14 includes a camera (not shown) forimaging the hands of the wearer. The gesture determination module 22 isconfigured to analyze the imaging of the hands of the wearer todetermine hand gestures and to correspondingly provide gesture controldata 82 that is processed by processing system 36 and used by processingsystem 36 and searchlight controller 42 to determine searchlight controldata 64 for controlling one or more features of searchlight 12. Forexample, gestures may be defined by gesture determination module forselecting operating parameters of searchlight 12 such as filter orfocus. In other embodiments, gaze control mode may be exited and gesturecontrol mode may be entered such that direction of beam 60 is controlledthrough gesture control. Gesture control data 82 may be used to controldisplay by headset display 16. For example, a zoom function may beinvoked by gesture control, which can be performed via a zoom of camera46 or through image processing by augmented/virtual reality generationmodule 64. In some embodiments, a field of view shown by headset display16 is controlled by a combination of gestures and head movements wherebycoarse control is achieved through gestures (e.g. by gesture selecting aparticular part of a currently displayed scene such as a floor of abuilding) with head movements providing detailed control of which partof the scene is centrally displayed. Alternatively, head motion controlmode can be exited and a gesture control mode can be entered in whichfield of view in headset display 16 is controlled primarily throughgesture control.

In accordance with various embodiments, searchlight control system 10includes a display device 24 including an interactive map module 26 anda touchscreen control module 28. Interactive map module 26 is configuredto generate an interactive map of the search scene and to synchronouslydisplay live images from camera 46. Although display device 24 is shownas a separate display in FIG. 1 , display device 24 can, in someembodiments, be headset display 16. That is, wearer of headset displaycan select an interactive map mode (e.g. through hand gestures andselectable options displayed through headset display 16) tosynchronously show live images or augmented/virtual reality live imagesand an interactive map. In either implementation, interactive map willcover a greater area than live images shown on display device 24.Display device 24 includes a touchscreen control module 28 configured toreceive touchscreen inputs through display device 24 on interactive mapand to output corresponding aerial vehicle commands 84. Autopilot system80 of aerial vehicle 40 is configured to respond to aerial vehiclecommands 84 to relocate aerial vehicle to a position selected oninteractive map. A similar function can be performed when interactivemap is generated on headset display 16, with selection of an area ofinteractive map being performed through gesture control. Touchscreencontrol module 28 is further configured to receive touchscreen inputsassociated with displayed live images to perform fine adjustment onposition and orientation of searchlight 12 and other searchlightfunctions (such as focus and filters). In some embodiments, displaydevice 24 and headset display 16 operate in parallel whereby a headsetcontrol mode is exited and a touchscreen control mode is entered suchthat searchlight 12 is primarily controlled through display device 24(including coarse interactive map touch searchlight control and finelive images searchlight control).

In accordance with various embodiments, searchlight control system 10includes a voice recognition unit 30 including a microphone 88 and avoice recognition module 32. Microphone can be included in headset 14 orprovided as a separate device. Voice recognition module 32 is configuredto process audio data received from microphone 88 to recognizevocabulary for which the voice recognition module 32 is trained orotherwise specially configured using known speech recognitionalgorithms. In some embodiments, vocabulary data 86 is received fromvocabulary database 34 such as a search and rescue focused vocabularydatabase 34. Voice recognition module 32 is configured to process theaudio data to identify any of a list of vocabulary in vocabulary data 86and to output corresponding voice control commands 90 to searchlightcontroller 42. Searchlight controller 42 is responsive to voice controlcommands 90 to control at least one function of the searchlight 12including direction of beam 60, filter, focus and other functions.Exemplary voice commands to be included in vocabulary data 86 includeany one or a combination of:

-   -   Searchlight movement commands        -   such as Turn, Tilt, Move, Rotate    -   Searchlight property commands        -   such as Increase/reduce brightness, Focus, Slave to camera    -   Filter selection Commands        -   such as Select Filter    -   Tracking commands        -   such as Track object (Human/vehicle) in focus, Record

In embodiments, voice recognition unit 30 is configured to receive voicecommands relating to display shown by headset display 16 including zoomand camera aiming functions, which can be executed throughaugmented/virtual reality generation module 64 or through cameracontroller 46.

In embodiments, voice control through voice recognition unit 30 is usedin association with gaze and head movement controls of headset 14. Thatis, some functions of searchlight 12 are controlled through voicecontrol commands 90 and some functions are controlled through gaze andhead tracking data 72. For example, movement of beam 60 is controlledbased on gaze and head tracking data 72 and filter selection, trackingor search light property commands are controlled through voice controlcommands 90. In other embodiments, a combination of voice, gesture andgaze/head movement controls is provided. For example, searchlightmovement commands are controlled based on gaze and head tracking data 72and voice control commands 90 and gesture control data 82 are used tocontrol different functions selected from searchlight property commands,filter selection command and tracking commands.

Various modes have been described herein including gaze control mode,gesture control mode, head motion control mode, headset control mode andtouchscreen control mode. These modes may be selected by voice, gestureor touch control. For example, a gesture may be defined to open a menuof selectable modes that is viewed through headset display 16. In thisway, a wearer of the headset 14 can transition between controlling adirection of searchlight 12 by gaze control and touchscreen control.Similarly, position of aerial vehicle 40 can be controlled throughgesture control or touchscreen control of interactive map depending onthe selected mode. When the modes are not in conflict with each other,some searchlight controls can be controlled by at least two of voicecontrol, gesture control and touchscreen control (e.g. searchlightfocus). Similarly, depending on the selected mode, field of view inheadset display is controlled by head motion control mode, gesturecontrol mode or touchscreen control mode. The present disclosure thusprovides multimodal control of a searchlight 12, a camera 46 (or atleast field of view in headset display 16) and optionally an aerialvehicle 40 through any combination of gesture, voice, gaze, head motionand touchscreen inputs.

In the embodiment of FIG. 1 , headset 14, display device 24, processingsystem 36 and voice recognition unit 30 are shown to be separate fromaerial vehicle 40. However, it is envisaged that one or more of thesecomponents are located in aerial vehicle 40. In other embodiments, thecomponents are ground based or some are ground based and some are basedin the aerial vehicle 40. For example, in the case of an unmanned drone,a ground-based mission coordinator may wear headset 14 and have accessto optional touchscreen display device 24 and optional voice recognitionunit 30. In other embodiments, a mission coordinator is located on theground and a pilot is located in the aerial vehicle 40. For example, themission coordinator may operate display device 24 to change certainfunctions of the searchlight 12 and to provide a request to the pilot(or the autopilot system 80) in the aerial vehicle 40 to change alocation of the aerial vehicle 40 using the interactive map. In yetfurther embodiments, two mission controllers are located on the ground,one operating the display device 24 and the interactive map and theother wearing the headset 14. The coordinator wearing the headset 14 isable to control a part of the scene being viewed and the direction ofthe searchlight 12 through head movement and gaze control, whilst thecoordinator operating the interactive map is able to control otherfunctions of searchlight and position of the aerial vehicle 40 throughthe interactive map.

FIG. 2 illustrates a flowchart of a method 200 of operating asearchlight control system 10, in accordance with various exemplaryembodiments. The various tasks performed in connection with method 200may be performed by software (e.g. program instructions executed by oneor more processors), hardware, firmware, or any combination thereof. Forillustrative purposes, the following description of method 200 may referto elements mentioned above in connection with FIG. 1 . It should beappreciated that method 200 may include any number of additional oralternative tasks, the tasks shown in FIG. 2 need not be performed inthe illustrated order, and method 200 may be incorporated into a morecomprehensive procedure or process having additional functionality notdescribed in detail herein. Moreover, one or more of the tasks shown inFIG. 2 could be omitted from an embodiment of the method 200 as long asthe intended overall functionality remains intact. Method 200 isdescribed in terms of being performed by one or more processors 48 ofprocessing system 36 executing instruction of computer program 50 storedon non-transitory memory. Processing system 36, processors 48 andcomputer program may be distributed differently to that shown in FIG. 1so as to be included in any one or more of headset 14, display device24, voice recognition unit 30, aerial vehicle 40 or elsewhere.

In step 210, processing system 36 receives video data 56 captured bycamera 46 mounted to aerial vehicle 40. Camera 46 is configured tocapture video of scene 52, which is lit by searchlight 12 and whichpotentially includes a target of the search mission (such as a human orvehicular target). In step 220 terrain data 58 defining syntheticterrain features (and their location in the world) is received byprocessing system 36 in addition to optionally receiving further datadefining further synthetic features such as graphical representations ofcertain objects (e.g. humans or vehicles). In step 230, processingsystem 36 is configured to generate, via augmented/virtual realitygeneration module 64, an augmented/virtual reality scene defined bydisplay data 70. Augmented/virtual reality generation module 64simulates binocular vision by generating different images (included indisplay data 70) to each eye, giving the illusion that a two-dimensionalpicture is a three-dimensional environment.

In step 240, the augmented/virtual reality scene defined in display data70 is displayed through headset display 16. A wearer of headset 14 isable to look up and down, left and right to traverse a view of theaugmented/virtual reality scene. Eye (gaze) and head movements aretracked in step 250 by headset gaze camera 20 and headset motion sensors18 to generate gaze and head tracking data 72, which is fed back toprocessing system 36 to update a position of a field of view defined bydisplay data 70. The field of view of scene 52 may be changed byaugmented/virtual reality generation module 64 sending a differentportion of a buffered augmented/virtual reality scene or by adjustingfocus and/or aiming of camera 46 by instructing camera controller 46 bysending scene capture data 76. Furthermore, a function of searchlight 12is controlled based on gaze and head tracking data 72 in step 270. Thatis, a gaze slaving algorithm is implemented so that an area of interestaccording to what the eyes of the wearer of headset 14 are looking at inimage space is transformed to an area of illumination by searchlight 12in the real world. Searchlight controller 42 calculates searchlightcontrol data 64 so that searchlight 12 illuminates that area of interestin the real world. In embodiments, the calculation is based on thetarget real world coordinates, the global position and orientation ofthe aerial vehicle 40 and searchlight properties (e.g. position onvehicle and current orientation) to determine an angular (and possiblyfocus) adjustment required to illuminate the target area.

In step 270 at least one additional mode of input is received forcontrolling the searchlight 12 and/or the aerial vehicle 40. Inembodiments, the additional mode of input is gesture control. Gesturesof hands of a wearer are detected by gesture determination module 22processing vision images of a wearer's hands (from an outwardly facingcamera of headset 14) or by detecting a constellation of light emittingdevices on hand controllers and using a constellation detectionalgorithm. Gesture determination module 22 outputs gesture control data82 to output one or more functions of searchlight 12 such as brightness,focus and type of filter being used. Various filters are possibleincluding different color filters and infrared filters (multispectralfilters), smoke/fog filters (for optimal use in fog or smokeconditions—generally an amber filter), a peripheral vision filter havinga band of light extending around a center spot, etc. Searchlight 12 mayinclude a filter wheel and brightness and/or focus setter that isoperated by searchlight actuators 62 under instruction from searchlightcontroller 42 based on gesture control data 82.

Another mode of input according to step 270, which may be additional toor alternative to gesture control, is voice control. Voice recognitionunit 30 detects voice commands in audio data received from microphone 88(which may be part of headset 14) and generates corresponding machinevoice control commands 90 to control a function of searchlight 12. Voicecontrol commands 90 are processed by searchlight controller 42 to outputsearchlight control data for implementing the commanded function. Forexample, brightness, focus and filter selection may be controlled byvoice control commands 90. Voice control commands 90 may also invoke atracking function of searchlight controller 42 (and camera controller46) whereby a target in augmented/virtual reality display scene shown inheadset 14 is identified and the target is automatically tracked bysearchlight 12 (and camera 46) by way of known tracking algorithms. Thetarget may be identified by a combination of gaze being directed at thetarget in the augmented/virtual reality display scene paired with atracking voice command. Processing system 36 is, in such an embodiment,configured to combine identified target in tracking data 72 and trackingcommand in voice control commands 90 and to send a searchlight trackinginvocation command to searchlight and camera controllers 42, 46.

Another mode of input according to step 270, which may be additional toor alternative to gesture and voice control, is touchscreen controlwhereby searchlight controls (e.g. searchlight movement commands,searchlight property commands, filter selection commands and trackingcommands) and/or aerial vehicle controls are selected by a user throughdisplay device 24. Display device 24 may be operated by a differentsearch coordinator from headset 14. As such, processing system 36 mayinclude rules to arbitrate control commands of searchlight 12,particularly if any controls from different modes of input are inconflict with one another. Different modes of operation may be set byusers to avoid control conflicts such as selecting a gaze control modeor a touchscreen control mode). In embodiments, display device 24 isconfigured to display an interactive graphical map by interactive mapmodule 26. A user may select an area of interest of interactive map,which is resolved by touchscreen control module 28 into aerial vehiclecommands 84. Interactive map will be of such a scale as to encompass aregion that is significantly outside of the range of view of camera 46.Interactive map provides a convenient way to command aerial vehicle 40to fly to the selected area of interest and to illuminate the area ofinterest with searchlight 12. Interactive map displays a region aroundaerial vehicle 40 as well as a graphical indicator of current vehiclelocation, which is based on global position data obtained from sensorsystem 68. At the same time as displaying interactive map, displaydevice 24 displays live video (or augmented video as described elsewhereherein) of the scene 52, thereby providing an overview of location of asearch beam 60 via the interactive map and also the actual area ofillumination of the search beam 60 in the video feed. In otherembodiments, the interactive map is shown on command by the headsetdisplay 16 at the same time as the augmented/virtual reality scene. Acommand for the interactive map can be selected by a wearer of theheadset through gesture or voice control. Further, the interactive mapcan be selected by the wearer through gesture interactions with the mapwhereby a pointer or other selection device shown in the virtual worldis used to select an area of interest in the interactive map.Alternatively, an area of interest in the virtual interactive map may beselected by gaze detection or by a combination of gaze and voicecommands. Headset 14 can respond by issuing aerial vehicle commands toaerial vehicle 40 in the same way as when selections are made on theinteractive map through display device 24 (e.g. by transforming imagespace coordinates to real world coordinates for use by autopilot system80).

The present disclosure thus allows an intuitive, immersive experiencefor controlling searchlight and aerial vehicle during a search andrescue mission. Accuracy and speed of control of searchlight controlsystem is significantly enhanced as compared to prior art hand gripcontrols, allowing the search coordinator to focus wholly on the searchtask.

While at least one exemplary embodiment has been presented in theforegoing detailed description, it should be appreciated that a vastnumber of variations exist. It should also be appreciated that theexemplary embodiment or embodiments described herein are not intended tolimit the scope, applicability, or configuration of the claimed subjectmatter in any way. Rather, the foregoing detailed description willprovide those skilled in the art with a convenient road map forimplementing the described embodiment or embodiments. It should beunderstood that various changes can be made in the function andarrangement of elements without departing from the scope defined by theclaims, which includes known equivalents and foreseeable equivalents atthe time of filing this patent application.

What is claimed is:
 1. A searchlight control system for an aerialvehicle, comprising: a searchlight mounted to the aerial vehicle; aheadset including a headset display and headset sensors; a searchlightcontroller in operable communication with the searchlight, the searchlight controller configured receive tracking data and to outputinstructions for controlling at least one function of the searchlightbased on the tracking data; and at least one processor in operablecommunication with the headset and the searchlight controller andconfigured to execute program instructions, wherein the programinstructions are configured to cause the at least one processor to:receive camera images of a scene including a portion being lit by thesearchlight; receive terrain data from a terrain database; generate anaugmented/virtual reality scene based on the camera images and theterrain data; command the headset display to display theaugmented/virtual reality scene; track gaze and head movement using theheadset sensors; and output the tracking data based on tracked gaze andhead movement, wherein the at least one function of the searchlightincludes one or more of: searchlight movement, searchlight property,filter selection and tracking, wherein searchlight movement includes oneor more of position, turn, tilt, move and rotate, searchlight propertyincludes one or more of adjust brightness, focus and slave to camera andtracking includes one or more of track object in focus and record. 2.The searchlight control system of claim 1, wherein a field of view ofthe augmented/virtual reality scene is adjusted based on the trackingdata.
 3. The searchlight control system of claim 1, comprisinggenerating camera control commands to adjust a field of view of thecamera based on the tracking data.
 4. The searchlight control system ofclaim 1, wherein the at least one processor is configured to execute theprogram instructions to generate gesture control data based on detectedhand gestures of the wearer of the headset and to control at least onefeature of the searchlight based on gesture control data.
 5. Thesearchlight control system of claim 1, wherein the at least oneprocessor is configured to execute the program instructions to generategesture control data based on detected hand gestures of the wearer ofthe headset and to automatically control a location of the aerialvehicle based on the gesture control data.
 6. The searchlight controlsystem of claim 1, wherein the at least one processor is configured toexecute the program instructions to receive a user input on aninteractive map that is displayed remotely from the aerial vehicle andto adjust a location of the aerial vehicle based thereon.
 7. Thesearchlight control system of claim 1 wherein the at least one processoris configured to execute program instructions to: command the headsetdisplay to synchronously display the augmented/virtual reality scene andan interactive map; receive data representative of interactions with theinteractive map; and automatically control a position of the aerialvehicle based on the interactions.
 8. The searchlight control system ofclaim 1, wherein the searchlight controller is configured to outputinstructions for controlling at least one function of the searchlightbased on a combination of at least two of the tracking data, gesturecontrol data, voice control data and touchscreen control data.
 9. Asearchlight control system for an aerial vehicle, comprising: asearchlight mounted to the aerial vehicle; a headset including a headsetdisplay and headset sensors; a searchlight controller in operablecommunication with the searchlight, the search light controllerconfigured receive tracking data and to output instructions forcontrolling at least one function of the searchlight based on thetracking data; and at least one processor in operable communication withthe headset and the searchlight controller and configured to executeprogram instructions, wherein the program instructions are configured tocause the at least one processor to: receive camera images of a sceneincluding a portion being lit by the searchlight; receive terrain datafrom a terrain database; generate an augmented/virtual reality scenebased on the camera images and the terrain data; command the headsetdisplay to display the augmented/virtual reality scene; track gaze andhead movement using the headset sensors; and output the tracking databased on tracked gaze and head movement, wherein the searchlight controlsystem further comprises a voice recognition unit configured to: receiveaudio input including a searchlight control command; process the audioinput to recognize the searchlight control command using trainedvocabulary in a speech recognition module; and output controls to thesearchlight controller based on the recognized searchlight controlcommand.
 10. The searchlight control system of claim 9, wherein thetrained vocabulary includes at least one of searchlight movementcommands, searchlight property commands, searchlight filter selectioncommands and tracking commands.
 11. A method of operating a searchlightcontrol system for an aerial vehicle, the method comprising: receiving,via at least one processor, camera images of a scene including a portionbeing lit by a searchlight mounted to the aerial vehicle; receiving, viathe at least one processor, terrain data from a terrain database;generating, via the at least one processor, an augmented/virtual realityscene based on the camera images and the terrain data; displaying, on aheadset display, the augmented/virtual reality scene; tracking gaze andhead movement using headset sensors; outputting tracking data based ontracked gaze and head movement; outputting, via the at least oneprocessor, instructions for controlling at least one function of thesearchlight based on the tracking data; and receiving, via the at leastone processor, a user input to an interactive map that is displayedremotely from the aerial vehicle and automatically adjusting, via the atleast one processor, a location of the aerial vehicle based thereon. 12.The method of claim 11, wherein a field of view of the augmented/virtualreality scene is adjusted based on the tracking data.
 13. The method ofclaim 11, comprising generating, via the at least one processor, gesturecontrol data based on detected hand gestures of the wearer of theheadset and controlling, via the at least one processor, at least onefeature of the searchlight based on gesture control data.
 14. The methodof claim 13 comprising automatically controlling, via the at least oneprocessor, a location of the aerial vehicle based on the gesture controldata.
 15. The method of claim 11, wherein the at least one function ofthe searchlight includes one or more of: searchlight movement,searchlight property, filter selection and tracking, wherein searchlightmovement includes one or more of position, turn, tilt, move and rotate,searchlight property includes one or more of adjust brightness, focusand slave to camera and tracking commands includes one or more of trackobject in focus and record.
 16. The method of claim 11, comprising:synchronously displaying, on the headset display, the augmented/virtualreality scene and an interactive map; receiving, via the at least oneprocessor, user interactions with the interactive map; and automaticallycontrolling, via the at least one processor, a position of the aerialvehicle based on the interactions.
 17. The method of claim 11,comprising: receiving, the at least one processor, audio input includinga searchlight control command; processing, via the at least oneprocessor, the audio input to recognize the searchlight control commandusing trained vocabulary in a speech recognition module; and outputting,via the at least one processor, controls to the searchlight controllerbased on the recognized searchlight control command.
 18. The method ofclaim 11, comprising outputting instructions, via the at least oneprocessor, for controlling at least one function of the searchlightbased on a combination of the tracking data and at least one of: gesturecontrol data, voice control data and touchscreen control data.